Description

Related to: #1991

This PR addresses issues with SQLite database locking in Azure Machine Learning (AML).

Reading online other users have also had this problem. It stems from the fact that in Azure ML the /home directory is mounted as CIFS filesystem which can't deal with SQLite3 lock. Read more here: https://stackoverflow.com/a/54051016

To resolve this, we can enable Write-Ahead Logging (WAL) mode to fix these locking issues.

We can either:

1. Adopt WAL for all environments: If we go with this option the session_store.db file will have a small change in its internal structure to indicate that it is now using WAL mode. This change is stored in the database's header.

2. Enable WAL mode conditionally for Azure ML: So we activate WAL mode only when the code detects that it is running in an AML environment. (Currently implemented in this PR).

Development notes

• Included PRAGMA journal_mode=WAL; to enable WAL mode.

• Implemented a condition to detect when it is running in an Azure Machine Learning environment using environment variables specific to Azure:AZUREML_ARM_SUBSCRIPTION, AZUREML_ARM_RESOURCEGROUP, or AZUREML_RUN_ID.

Provided two options:

1. Apply WAL mode universally across all environments.
2. Apply WAL mode only when the AML environment is detected, using the is_azure_ml flag.

QA notes

I tested this in Azure Machine Learning studio and confirmed that the database operates in WAL mode without the locking issues.

Tested locally as well it works as expected with and without enabling WAL mode.

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added new entries to the RELEASE.md file
• Added tests to cover my changes